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Abstract 

Deception  aims  to  affect  a  decision-making  process  in  a  way  that  somehow 
benefits  the  deceiver.  More  knowledge  is  desired  in  the  area  of  purely  text-based 
scenarios.  Thirty-seven  graduate  students  at  the  Air  Force  Institute  of  Technology 
participated  in  an  experiment  in  order  to  gain  an  initial  understanding  of  how  people 
determine  whether  text  has  been  manipulated,  and  to  identify  specific  areas  that  may  be 
more  closely  investigated  in  future  research.  Excerpts  were  drawn  from  editions  of  a 
current-events  newsletter  that  the  participants  receive  on  a  weekly  basis  as  part  of  their 
enrollment  in  the  graduate  program.  Some  of  the  excerpts  were  manipulated,  and  others 
were  not.  Participants  were  shown  a  set  of  these  excerpts  and  were  asked  to  give  ratings 
of  their  perceived  familiarity  with  the  subject  of  the  excerpt,  their  perception  of  the 
relevance  of  that  subject  to  themselves,  and  their  perception  of  whether  they  believed  that 
the  excerpt  had  been  manipulated.  As  part  of  their  manipulation  answers,  participants 
were  allowed  to  indicate  heightened  confidence  in  their  answers  by  selecting  a  version  of 
the  answer  that  included  the  word  “definitely.”  Analysis  of  the  responses  showed  support 
for  associations  between  familiarity,  relevance,  and  definite  answers.  Analysis  further 
showed  support  for  an  association  between  definite  answers  and  increased  rate  of 
accuracy  in  determining  whether  an  excerpt  was  manipulated.  The  analysis  did  not  show 
support  for  an  association  between  familiarity  and  accuracy,  nor  did  it  show  support  for 
an  association  between  relevance  and  accuracy. 
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TEXT  MANIPULATION  JUDGMENT  ACCURACY:  AN  EXPLORATORY 

STUDY 


Chapter  1  -  Introduction 


Introduction 

When  two  or  more  parties  compete  “to  reach  or  obtain  something  that  only  one 
can  possess”,  we  may  say  that  these  parties  are  rivals.  Merriam-Webster’s  Dictionary 
(www.m-w.com,  2003)  gives  multiple  definitions  for  the  word  “competition”;  they  may 
be  combined  as  “the  effort  of  two  or  more  parties  acting  independently... for  some 
environmental  resource  in  short  supply.” 

When  a  party  decides  on  a  course  of  action  to  reach  its  objectives,  it  carries  out  a 
decision-making  process.  This  process  involves  an  overall  objective,  information  about 
the  matter  at  hand  (including  environmental  conditions),  a  process  for  determining  what 
choices  are  included  in  the  set  of  alternatives,  what  the  likely  outcome  of  selecting  a 
given  alternative  is,  and  criteria  for  selecting  one  of  the  alternatives 
(http://pespmcl.vub.ac.be/ASC/DECISI_THEOR.html).  Much  of  the  above  enters  the 
decision-making  process  as  information.  (Contentions  between  semantic  differences  of 
the  words  “data”,  “information”,  and  “knowledge”  are  not  addressed  here;  for  purposes 
of  discussion  within  this  document,  the  word  “information”  is  used  to  cover  all  of  them.) 
Once  the  decision  is  made  and  implemented,  actual  results  (which  may  or  may  not  match 
the  expected  results)  will  be  obtained. 

Concerning  the  Merriam- Webster  phrase  “acting  independently”;  we  may 
intuitively  infer  that  each  party  independently  makes  the  decisions  that  guide  its  own 
actions.  However,  it  is  often  more  advantageous  to  influence  an  opponent’s  decision 
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process  as  a  means  of  influencing  his  actions.  Done  honestly,  this  may  involve  sharing 
accurate  information  with  that  competitor,  which  hopefully  will  influence  his  decision 
making  process  in  a  beneficial  way.  On  a  practical  basis,  this  is  most  likely  to  occur 
when  the  results  are  seen  to  be  mutually  beneficial.  If  the  same  approach  were  taken 
while  using  inaccurate  information  instead  of  accurate  information,  we  would  call  this 
deceit. 

Since  information  is  obtained  and  shared  by  many  means,  deceit  can  also  occur  by 
many  means.  The  simplest  fonn  of  this  and  arguably  the  most  thoroughly  studied  so  far, 
is  in  the  realm  of  verbal  communication  between  one  person  and  another.  Simple  deceit 
in  this  case  would  consist  of  a  person  telling  a  lie.  Other  modes  that  are  currently  under 
study  involve  various  modes  of  communications,  mostly  via  media  such  as  video-  and 
audio-taped  messages,  and  computer  or  information  systems,  and  the  implications  that 
those  alternate  modes  entail. 

In  more  complex  scenarios,  such  as  those  that  are  carried  out  by  military  forces 
while  conducting  warfare,  characteristics  of  environment  may  be  changed  so  that  the 
opponent  will  perceive  and  use  inaccurate  information  (Biros,  et  al,  2002).  The  objective 
is  to  purposely  enter  inaccurate  infonnation  into  the  enemy’s  decision-making  process,  to 
therefore  alter  the  decision  that  is  made,  and  to  in  turn  alter  the  real  world  results.  The 
effectiveness  of  the  deception  may  be  evaluated  by  comparing  the  results  (or  expected 
results)  of  the  non-deception  scenario  with  those  of  the  scenario  where  deception  was 
carried  out. 

In  essence,  deception  differs  from  the  “independent  action”  that  describes  pure 
competition.  The  deceiver  depends  on  the  victim’s  manipulated  decision  process  to 
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produce  beneficial  decisions  and  subsequent  beneficial  results.  Below  are  some 
examples  of  military  deceptions. 

One  biblical  example  in  the  Bible’s  Old  Testament  Book  of  Judges,  which 
chronicles  a  vastly  inferior  Israeli  force  (300  men)  led  by  Gideon  deceiving  an  enemy 
(described  as  a  countless  number  of  people  that  filled  a  valley)  by  arriving  at  night  and 
displaying  torches  in  such  a  way  that  the  comparatively  miniscule  Israeli  army  appeared 
to  be  much  larger  than  that  of  its  enemy.  The  enemy  perceived  that  they  were  vastly 
outnumbered  on  all  sides  and  decided  to  surrender  to  Gideon  and  his  Israeli  force.  Had 
the  armies  actually  fought,  the  Israelis  most  likely  would  have  been  slaughtered 
(http://www.carm.org/kiv/Judges/Judges  7.htm).  This  action  (the  deceptive  display) 
provided  a  direct  result  (surrender)  to  Gideon. 

Another  military  ruse  was  the  Allies’  now-famous,  and  well-published,  left  hook 
maneuver  in  the  Gulf  War.  General  Schwarzkopf  feigned  an  amphibious  attack  in  the 
East;  Iraq  took  the  appearance  of  the  feint  into  account  in  its  defense  decision-making 
and  decided  to  place  forces  in  the  East  to  face  the  feigned  attack.  This  gave  conditions  of 
reduced  defense  in  the  West  and  enabled  the  Allied  Force’s  left  hook  to  be  more  easily 
executed  (Hines).  This  example  demonstrates  a  case  of  competitive  advantage.  The  feint 
did  not  provide  an  immediate  result  (surrender)  however  it  did  set  the  stage  for  future 
action  where  the  Allied  Force  was  at  a  great  advantage  (Iraqi  defenses  were  low  and 
more  easily  overcome  in  the  West  where  the  left  hook  was  executed). 
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Problem  Statement 


As  described  in  the  introduction,  deception  can  be  seen  as  a  significant  tool  from 
the  deceiver  perspective,  or  as  a  grave  threat  from  the  victim’s  perspective.  The 
implications  can  range  from  “no  effect”  to  a  critical  factor  of  success  in  important  events, 
such  as  warfare.  While  the  art  of  deceit  has  been  practiced  for  centuries,  the  scientific 
study  of  deception  has  comparatively  just  begun.  The  study  of  this  area  draws  upon 
expertise  in  numerous  disciplines  (Biros,  et  al,  2002),  and  must  deal  with  diverse 
scenarios  which  can  be  quite  complex  even  in  their  simplest  forms. 

So  far,  a  preponderance  of  work  has  dealt  with  interpersonal  deception  scenarios 
where  individuals  are  conversing  in  direct  contact  with  one  another  (Buller  and  Burgoon, 
1996;  Burgoon,  et  al,  1996;  Decaire,  2000;  DePaulo  and  DePaulo,  1989;  Vrij,  2000). 
Potential  nonverbal  cues  are  considered  to  be  abundant  under  such  a  scenario.  However, 
other  forms  of  communication,  and  the  detection  of  deception  within  them  have  shown  to 
be  more  challenging  to  research,  and  have  been  subject  to  less  research  so  far.  Some  on¬ 
going  efforts  are  beginning  to  cover  deception  that  may  occur  in  media  such  as  video, 
audio  and  via  infonnation  systems  (Biros,  et  al,  2002;  Horn,  2001).  One  area  that  still 
needs  research  is  the  area  of  text  information,  where  the  information  itself  is  emphasized 
over  the  mode  of  delivery  of  the  information.  This  study  aims  to  explore  this  vital  area. 

Research  Questions 

How  accurate  are  people  at  detecting  deception/manipulation  of  text  infonnation? 
What  are  some  preliminary  factors  that  affect  the  detection  of  deception  or  manipulation 
in  text  information? 
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Blueprint  for  This  Thesis 

To  answer  the  above,  an  experiment  was  performed  in  order  to  glean  preliminary 
information  about  the  detection  of  deception  in  text  infonnation.  The  goal  was  to  gain  a 
foothold  in  the  study  of  this  area,  and  to  provide  insight  into  what  topics  might  be  fruitful 
for  future  research. 

In  Chapter  2,  extant  literature  that  covers  the  current  theory  base  for  this  area  will 
be  reviewed.  In  Chapter  3,  the  methodology  used  in  the  design  of  this  study  and  the 
experiment  are  covered.  In  Chapter  4,  the  analysis  and  results  are  shown.  In  Chapter  5,  a 
review  of  the  findings  and  a  summary  of  what  has  been  learned  are  presented.  The  next 
chapter  will  cover  the  myriad  of  constructs  that  pertain  to  this  study. 
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Chapter  2  -  Literature  Review 


Overview 

As  mentioned  in  the  introduction,  the  study  of  deception  and  deception  detection 
draws  on  knowledge  in  a  wide  variety  of  disciplines  (Biros,  et  al,  2002).  Below,  the 
concepts  of  decision-making,  infonnation,  deception,  interpersonal  deception  theory, 
information  manipulation  theory,  data  quality,  data  relevance,  and  familiarity  will  be 
examined.  A  model  will  be  developed  that  encompasses  the  constructs  that  will  be  tested 
in  this  study;  this  model  will  lead  to  a  discussion  of  the  methodology  and  the  experiment 
that  was  performed. 

Decision-Making 

Decision-making  is  described  to  consist  of  the  formulation  of  information  about 
one’s  objectives  and  the  current  environment,  ascertaining  a  set  of  alternative  courses  of 
action  that  are  available  to  meet  those  goals,  estimation  of  what  the  consequences  or 
results  will  be  of  selecting  each  of  the  alternatives,  and  choosing  the  alternative  that  will 
most  likely  provide  the  most  benefit  to  the  decision-maker  once  the  chosen  course  of 
action  is  implemented  (http://pespmcl.vub.ac.be/ASC/DECISI_THEOR.html).  In 
essence,  decision-making  is  about  estimating  the  future  results  of  a  set  of  alternative 
courses  of  action,  and  selecting  the  one  that  is  most  likely  to  be  the  most  beneficial  to  the 
decision-maker.  Decision-making  perfonnance  is  hindered,  however,  by  information 
processing  limitations  of  humans  as  decision-makers. 

Eppich  (2001)  describes  a  progression  of  decision  making  theory  from  its  early 
20th  Century  concepts  of  an  idealistic  and  well-defined  process,  also  known  as  Subjective 
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Expected  Utility,  to  a  more  uncertainty  and  reality-based  process  where  objectivity  gives 
way  to  heuristics.  Situations  exist  in  which  the  decision  maker  is  unable  to  identify  the 
outcome  with  100  percent  confidence  (Saks  and  Kidd,  1986;  Eppich,  2001).  Eppich 
points  out  that  people  often  are  not  able  to  process  all  of  the  information  related  to  the 
specifics  of  the  decision  alternatives  and  consequences  that  a  fully  objective  and  100 
percent  confident  decision  might  require.  The  use  of  heuristics  helps  a  decision-maker  to 
simplify  the  decision  process  by  reducing  the  amount  of  information  that  is  actually 
processed. 

The  information  that  enters  the  decision  process  may  possibly  be  incomplete  or 
inaccurate.  Perfecting  the  infonnation  that  is  input  to  the  process  can  prove  to  be 
difficult,  and  a  military  adage  estimates  that  decisions  are  often  made  with  only  forty  to 
seventy  percent  of  the  information  that  a  decision-maker  would  like  to  have.  In  addition, 
the  estimates  of  the  consequences  for  each  alternative  may  be  uncertain  and/or  inaccurate 
as  well.  Uncertain  consequences  can  be  evaluated  in  a  heuristic  and/or  probabilistic 
manner  to  detennine  which  alternative  offers  the  most  utility  to  the  decision  maker 
(Principia  Cybernetica  Web).  It  is  also  possible  that  an  optimal  decision  may  later  be 
poorly  implemented,  and  therefore  may  produce  poor  results.  All  of  the  above  can 
synergistically  interact  to  degrade  the  effectiveness  of  a  given  decision. 

Decision-making  processes  may  vary  in  tenns  of  time,  resources  and  effort 
required  to  accomplish  them,  the  amount  of  each  depends  on  the  scenario  and  the 
decision  that  is  to  be  made.  In  the  text  information-based  scenario  that  this  study 
explored,  however,  the  focus  was  on  the  information  directly.  It  seeks  to  measure  some 
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characteristics  of  how  a  person  approaches  the  decision  about  whether  text  infonnation  is 
manipulated  or  not. 

Information 

Zmud  (1990)  describes  an  organization’s  members  as  consumers,  managers  and 
purveyors  of  information.  More  specifically,  he  describes  information  as  meaning  that 
humans  assign  to  items  or  data,  and  that  meaning  is  used  to  make  decisions. 

Data  fusion  is  a  concept  that  takes  this  process  further  in  that  it  attempts  to  bring 
together  and  organize  infonnation  from  multiple  sources  and  that  is  available  in  different 
formats  in  a  way  that  supports  decision-making  in  complex  and  dynamic  environments, 
such  as  military  command  and  control  (Bisantz,  et  al,  1999).  Data  systems  and  decision- 
aids  are  a  means  to  bring  infonnation  to  decision-makers  in  a  way  that  has  useful 
meaning.  Typical  automated  infonnation  systems  are  designed  to  provide  this  type  of 
support.  Increased  dependence  on  such  systems  may,  however,  increase  vulnerability  to 
“strategic  information  manipulation,”  which  is  what  Zmud  (1990)  calls  the  practice  of 
purposely  implementing  inaccurate  data  (Biros,  2002). 

Deception 

Deception  is  commonly  defined  as  a  message  knowingly  transmitted  by  a 
deceiver  to  foster  a  false  belief  or  conclusion  by  the  receiver  (Burgoon,  et  al,  1996).  In 
addition,  when  a  person  deliberately  tries  to  foster  in  others  a  belief  or  understanding 
which  he  considers  to  be  untrue,  that  person  is  engaging  in  deception.  (DePaulo,  et  al, 
1985). 
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Biros,  et  al,  (2002)  also  point  out  that  some  persons  have  been  motivated  to 
manipulate  information  to  influence  the  behavior  of  others,  and  that  this  is  a  complex 
phenomenon  that  draws  interest  from  a  number  of  fields  (Feldman  and  March  1981, 
Johnson,  et  al,  1993,  Miller  and  Stiff  1993,  Biros,  et  al,  2002).  Obviously,  purposeful 
placement  of  erroneous  information  is  one  means  of  deceiving. 

McComack  (1992)  points  out  that  individuals  may  deceive  as  a  means  of 
reconciling  the  interest  of  providing  information  that  a  receiver  desires  or  needs  (or  not 
intervening,  and  therefore  allowing  the  receiver  to  obtain  it),  and  the  competing  personal 
interest  that  would  be  damaged  if/when  the  receiver  obtains  this  information  (Bowers,  et 
al,  1977;  Turner,  et  al,  1975).  In  these  scenarios,  the  deceiver  may  produce  a  deception 
by  manipulating  information  and  conveying  it  to  the  receiver. 

Deception  Detection 

Several  studies  indicate  that  cues  exist  when  a  person  is  engaging  in  deceit,  and 
that  these  clues  may  be  detected  by  a  potential  victim.  Depaulo,  et  al,  (1985),  examine 
and  summarize  the  prospect  of  validating  information  with  an  experienced  person,  or  by 
seeking  inconsistencies  in  information  over  time.  These  clues  may  take  the  form  of 
behaviors  that  are  not  controlled  (that  is,  they  are  not  adapted  to  support  the  deception). 
The  term  that  is  often  used  in  this  research  is  that  a  cue  has  “leaked”  through  the  facade 
that  the  deceiver  is  attempting  to  achieve;  overall,  this  is  leakage  theory.  These  leaks 
may  offer  cues  that  a  deception  is  present. 

Davies  and  Tune  (1969)  describe  several  aspects  of  signal  theory,  which  seem  to 
be  particularly  germane  to  deception  detection.  Signal  theory  stipulates  that  in  order  for 
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something  in  an  environment  to  be  noticed,  it  must  be  differentiated  from  its  environment 
in  some  way.  Further,  this  different  characteristic  must  appear  at  a  high  enough  intensity 
for  a  sufficient  length  of  time  that  it  is  sensed. 

When  applied  to  deception  detection  efforts,  it  follows  that  a  cue  to  a  deceptive 
act  must  be  intense  and  of  enough  duration  to  be  noticed  by  an  observer,  or  potential 
deception  detector.  The  scenario  and  the  environment  in  which  it  commences  detennine 
what  specific  cues  might  indicate  that  a  deception  is  present.  In  interpersonal 
communication,  there  is  extant  reference  to  nonverbal  behavior  and  its  connection  to  the 
message  that  is  presented  (Decaire,  2000). 

Vrij,  et  al  (2000),  suggest  that  there  are  three  ways  to  detect  when  a  person  is 
engaging  in  deception;  those  ways  are:  (1)  observing  their  behavior,  (2)  analyzing  the 
content  of  the  message,  and  (3)  measuring  physiological  responses.  This  is  problematic 
in  some  scenarios,  because  these  were  meant  to  describe  interpersonal  communication 
where  there  is  a  potentially  rich  set  of  behavioral  characteristics  to  observe. 

In  media  where  the  observable  characteristics  of  a  communicative  process  are 
more  limited,  these  cues  may  differ,  or  may  assume  different  degrees  of  prominence 
(Horn,  2001).  In  cases  like  this  study,  where  text  information  is  of  interest  and  is 
examined,  (and  where  the  crafting  of  that  text  is  not  observable  by  the  detector)  only  the 
second  means  (analyzing  content)  seems  applicable.  In  regard  to  analyzing  content  to 
determine  if  deception  is  present,  Decaire  (2000)  summarizes  that  individuals  who  have  a 
baseline  reference  for  nonnative  information  (information  that  is  known  to  be  free  of 
deception)  available  to  them  are  significantly  better  able  to  accurately  detect  deception. 
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Interpersonal  Deception  Theory 

Interpersonal  Deception  Theory  (IDT)  proposes  that  infonnation  management 
plays  a  role  in  a  deceiver’s  strategy  to  deceive,  and  that  deceivers  systematically  alter 
information  when  creating  their  deceptive  message  (Burgoon,  et  al,  1996).  Turner,  et  al 
(1975),  divide  deception  into  two  main  areas:  distortions  and  concealments,  where 
distortions  tend  to  add  information  or  change  the  information  that  is  present,  and 
concealments  tend  to  omit,  reduce,  or  divert  attention  from  information.  Eckman  (1985) 
argues  that  concealments  alone  may  be  used  to  deceive;  however,  it  is  often  the  case  that 
a  combination  of  concealing  and  falsifying,  or  distorting,  is  utilized  to  successfully 
deceive.  The  experiment  that  has  been  carried  out  for  this  study  focuses  on  distortion  of 
existing  information,  and  in  agreement  with  Eckman,  some  original  portions  of  the 
information  were  replaced  or  concealed  in  order  to  support  the  deception. 

Deception  types  or  tactics  have  been  categorized  or  typified  in  a  number  of  ways. 
Burgoon,  et  al,  (1996)  chronicled  Turner’s  (1975)  deception  categories,  including  lies, 
exaggerations,  secrets,  half-truths,  and  diversionary  responses;  and  Hopper  and  Bell’s 
(1984)  six  communicative  forms  of  deception:  lies,  masks,  unlies  (false  implications), 
crimes,  fictions  and  playings  (all  of  these  six  are  common  English  language  labels  for 
acts  of  deception).  Burgoon,  et  al,  (1996)  point  out  that  these  various  forms  are  means  to 
understand  the  types  of  acts  that  occur;  Burgoon,  et  al  (1996),  also  point  out  that  these 
labels  are  not  all-inclusive  of  all  deceptive  acts,  and  some  acts  can  be  described  by  more 
than  one  of  these  categories. 

Burgoon,  et  al,  (1996)  propose  five  strategic  dimensions  in  which  a  deceiver 
might  alter  a  message  in  order  to  enact  a  deception.  They  are  completeness,  veridicality 
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(trustworthiness,  actual  or  apparent),  “directness  and  relevance,”  clarity,  and 
personalization.  Further,  the  receiver  of  a  communication  may  be  able  to  examine  it  to 
detennine  the  perception  of  a  discrepancy  along  these  dimensions  can  be  perceived, 
which  in  turn  might  indicate  a  deception. 

Taxonomy  of  Deception  Strategies 

Biros,  et  al  (2002),  utilized  information  manipulation  techniques  in  their 
experiment  that  were  based  on  the  taxonomy  of  deceptive  tactics  that  is  put  forth  by 
Johnson,  et  al  (1993).  Figure  1  lists  these  tactics  and  their  descriptions.  Masking  and 
inventing  were  used  in  the  experiment,  as  described  in  Chapter  3. 


Figure  1.  Taxonomy  of  Deceptive  Tactics  (from  Biros,  et  al,  2002;  based  on 
Johnson,  et  al,  1993) 


Tactic 

Description 

Masking 

Deleting  from  the  environment  attributes  that  suggest  the  correct  representation. 

Double  Play 

Manipulating  attributes  in  the  environment  in  a  way  so  as  to  weakly  suggest  the 
correct  representation.  The  purpose  is  to  reinforce  incorrect  representations  by 
weakly  suggesting  the  correct  one. 

Mimicking 

Modifying  attributes  in  the  environment  in  a  way  so  as  to  suggest  the  incorrect 
representation.  Essentially  suggestions  (not  necessarily  deceptions  in  and  of 
themselves)  are  included  to  support  the  incorrect  representation. 

Dazzling 

Modifying  attributes  in  the  environment  in  such  a  way  as  to  obscure  or  blur  those 
attributes  whose  interpretation  suggests  the  correct  representation  and  to  emphasize 
those  attributes  whose  interpretation  suggests  the  incorrect  one. 

Inventing 

Adding  new  attributes  to  the  environment  in  order  to  suggest  the  incorrect 
representation. 

Repackaging 

Modifying  attributes  in  the  environment  in  order  to  hinder  the  generation  of  the 
correct  representation.  Repackaging  is  weaker  than  mimicking  because  it  is  based 
on  justification  and  distortion  rather  than  replication  of  attributes. 

Decoying 

Adds  new  attributes  to  the  environment  in  order  to  hinder  identification  of  the 
correct  representation.  It  is  weaker  than  inventing  because  the  decoys  are  not 
directly  suggestive  of  the  incorrect  one.  It  simply  directs  attention  away  from  the 
correct  one. 
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Information  Manipulation  Theory 

The  English  language  philosopher  Paul  Grice  (1975)  proposes  that  in  ordinary 
conversation,  speakers  and  hearers  share  a  Cooperative  Principle.  Speakers  shape  their 
utterances  to  be  understood  by  hearers.  Grice's  Cooperative  Principle  is  comprised  of 
four  such  norms,  or  maxims:  (1)  Quality,  the  speaker  tells  the  truth  or  his  message  is 
provable  by  adequate  evidence;  (2)  Quantity,  the  speaker  is  as  informative  as  required  to 
convey  the  information;  (3)  Relation,  a  conversational  response  is  relevant  to  topic  of 
discussion;  and  (4)  Manner,  a  speaker  avoids  ambiguity  or  obscurity,  and  he  is  direct  and 
straightforward. 

McComack  (1992)  adapted  Grice’s  maxims  via  Information  Manipulation  Theory 
(IMT)  where  a  deceiver’s  messages  may  covertly  violate  Grice’s  principles  in  order  to 
reconcile  conflicts  between  the  deceiver’s  personal  goals  and  the  results  of 
allowing/providing  accurate  information.  The  emphasis  of  McComack’ s  work  is  the 
content  of  the  infonnation  (including  its  meaning  and  context)  is  key  in  enacting  a 
deception,  or  detecting  its  presence. 

In  contrast  to  the  focus  on  the  specific  deceptive  tactics  by  Johnson,  et  al  (1993), 
and  Burgoon,  et  al  (1996),  McComack  (1992)  reasons  that  if  a  deceptive  message  does 
not  violate  any  of  the  Cooperative  Principle  maxims,  that  a  deception  is  likely  to  go 
unnoticed  by  the  receiver.  That  is,  if  a  message  satisfies  a  receiver’s  cooperative 
expectations  (is  of  high  enough  quality,  contains  a  sufficient  quantity  of  information,  is 
relevant  to  the  receiver,  and  is  received  in  an  appropriate  manner)  it  is  likely  to  be 
accepted  by  the  receiver  as  true.  If  any  of  these  maxims  are  unmet,  then  the  message  is 
likely  to  be  rejected  by  the  receiver.  Note  that  this  interpretation  does  not  necessarily 
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imply  that  the  receiver  will  suspect  a  deception  per  se,  but  rather  that  it  may  be  ignored  or 
discarded. 

From  the  perspective  of  using  inaccurate  information  as  a  deception  tool,  we  must 
also  consider  inaccuracies  that  occur  without  a  deceptive  motive  behind  them,  and 
therefore  it  is  appropriate  to  address  data  quality. 

Data/Information  Quality 

Wang  and  Strong  (1996)  describe  four  characteristics  that  data  must  possess  in 
order  to  be  useful  to  a  decision-maker.  The  characteristics  are  accessibility, 
interpretability,  relevance  and  accuracy.  In  order  for  a  user  to  consider  the  data  to  be  of 
high  quality,  he  must  be  able  to  access  it  (be  aware  of  the  existence  of  the  infonnation 
and  knowing  how  to  get  to  it).  The  user  must  be  able  to  read  or  interpret  it  (for  instance, 
text  infonnation  should  be  in  a  language  or  style  that  is  understandable  to  the  user).  The 
user  must  find  the  information  to  be  helpful  in  the  task  he  is  trying  to  perform  (in  a 
decision-making  scenario  for  instance,  this  information  would  help  him  to  discriminate 
between  available  decision  alternatives  and  the  consequences  of  each).  Finally  the 
information  must  be  accurate  (the  user  must  believe  that  the  information  is  true,  or  he 
may  in  turn  simply  ignore  it).  Other  published  work  discusses  errors  in  stored  data. 

Laudon  (1986)  examined  a  criminal  justice  database  that  contained  records  about 
warrants.  This  particular  system  stood  alone,  however  the  information  within  it  was 
compared  to  other  records  to  find  discrepancies  and  reconcile  them.  Records  were  found 
to  contain  errors  in  several  areas,  including  infonnation  about  the  disposition  of  the 
warrants  (incomplete  records  mostly,  where  updates  to  clear  the  wanants  in  question 
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were  not  accomplished)  and  identities  of  persons  involved  in  the  warrants  (persons  who 
were  truly  not  the  subject  of  a  warrant  appeared  in  the  database  and  were  associated  with 
an  outstanding  warrant;  these  persons  were  vulnerable  to  have  false  warrants  exercised 
against  them  due  to  the  bad  information). 

Klein,  et  al  (1997a  and  1997b)  found  in  their  studies  that  organizational  databases 
can  contain  significant  rates  of  error  contained  within  them,  and  they  estimate  that  one  to 
ten  percent  is  a  typical  rate  of  error.  Further,  if  this  erroneous  data  is  used,  it  may  have 
adverse  impact  on  decisions  upon  which  it  is  based.  Klein,  et  al’s  (1997a)  experiment 
gave  support  that  when  tasked  to  find  errors  in  such  data,  that  participants  perfonned 
better  when  given  explicit  error  detection  goals.  Klein,  et  al,  also  said  that  error  detection 
ability  varies  under  different  detection  scenarios  and  circumstances,  and  that  some 
previously  inconclusive  studies  were  too  general  in  nature  to  find  this. 

Klein,  et  al  (1997b)  also  posits  that  the  payoff  for  detecting  an  error  in  the  data 
might  have  an  influence  on  the  effort  expended  in  order  to  find  an  error.  Three 
dimensions  that  are  described  are  materiality,  incentives,  and  ease  of  verification  and 
correction.  Materiality  deals  with  the  concept  that  the  effort  expended  in  finding  an  error 
relates  to  the  amount  of  negative  effect  that  error  could  cause  (i.e.  an  insignificant  effect 
may  not  make  it  worth  trying  very  hard  to  find  an  error).  Incentives  deal  with  what  will 
result  from  ignoring  the  error  vice  expending  effort  to  find  it  (i.e.  is  the  consequence  of 
the  error  worse  than  the  work  involved  in  trying  to  find  it?);  incentives  (or  disincentives) 
are  weighed  and  used  to  choose  whether,  and  to  what  level  of  effort  to  seek  the  error. 
Lastly,  ease  of  verification  and  correction  deals  with  the  concept  that  effort  expended  in 
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finding  an  error  will  vary  according  to  how  easy  it  is  to  confirm  the  error  is  in  deed  and 
error;  how  recognizable  the  error  is  likely  to  be. 

Self-efficacy 

Efficacy  is  simply  perceived  ability,  or  the  “power  to  produce  an  effect”  (www.m- 
w.com,  2003).  Self-efficacy  is  a  perception  about  one’s  own  ability  to  perfonn  a  task,  or 
confidence.  If  a  person  believes  they  are  able  to  accomplish  something,  their  self- 
efficacy  about  that  task  is  high.  Harrison,  et  al  (1997)  summarize  that  the  construct  of 
self-efficacy  is  positively  associated  with  actual  performance. 

Manipulation  Types 

For  purposes  of  this  study,  four  manipulation  types  were  identified  for  use  in  this 
experiment.  The  types  are  significant  value  manipulation,  order  of  magnitude  value 
manipulation,  logic  manipulation,  and  identity  reattribution. 

Within  the  work  of  Klein,  et  al  (1997b),  it  was  supported  that  a  higher  degree  of 
materiality  of  an  error  led  to  increased  accuracy  in  finding  that  error;  that  is,  errors  of 
greater  perceived  consequence  (such  as  an  erroneous  cost  difference  that  ultimately 
would  amount  to  a  significant  amount  of  money)  were  more  readily  discovered.  This 
concept  was  adapted  to  the  two  types  of  manipulation  used  within  this  study  that  dealt 
with  quantitative  values.  These  two  categories  were  a  significant  amount  (i.e.  values 
were  halved  or  doubled),  or  order  of  magnitude  (i.e.  values  were  multiplied  or  divided  by 
a  power  of  ten).  To  accomplish  this,  numerical  digits  were  manipulated,  or  quantitative 
words  were  changed  (i.e.  millions  instead  of  thousands,  or  none  instead  of  all). 
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As  stated  previously,  Laudon’s  (1996)  work  that  showed  that  errors  in  a  warrant 
record  system  often  conflicted  with  the  true  facts  of  a  warrant  (i.e.  a  warrant  was  shown 
to  be  open,  when  in  fact  it  was  closed),  or  persons  were  misidentified  with  a  warrant  (i.e. 
the  wrong  person  was  associated  with  a  given  situation).  In  keeping  with  Laudon’s 
findings,  some  information  within  this  experiment  was  manipulated  by  two  means:  the 
logic  and  supporting  statements  were  changed  (usually  so  that  the  manipulated  statement 
had  an  overall  opposite  meaning  to  the  original  statement),  and  actions  were  reattributed 
to  different  persons  (a  name  was  changed,  but  the  act  was  left  the  same). 

Relevance 

In  addition  to  Wang  and  Strong’s  (1996)  mention  of  relevance  as  a  requirement 
for  particular  data  to  be  perceived  to  have  high  quality,  McComack,  et  al,  (1992) 
manipulated  information  in  terms  of  relevance  (and  three  other  dimensions  including 
quality,  quantity  and  manner)  in  order  to  sense  differences  in  detection  accuracy  when 
these  different  modes  of  deception  were  employed. 

In  either  case,  relevance  is  determined  by  the  perspective  of  the  person  that  will 
use  the  infonnation.  For  example,  the  atomic  weight  of  uranium  is  of  high  relevance  to  a 
nuclear  physicist  in  his  work,  but  may  be  of  very  little  interest  to  a  fast  food  worker  is  his 
work. 

Familiarity 

Biros,  et  al,  (2002)  studied  a  construct  they  described  as  domain  experience, 
which  can  be  interpreted  to  mean  “gained  knowledge.”  Klein  (1996)  evaluated 
experience  in  regards  to  erroneous  and  manipulated  infonnation,  and  in  essence  found 
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that  the  level  of  experience  influenced  expectations  about  the  type  and  amount  of 
discrepancies  that  can  and  were  likely  be  found  in  a  data  store. 

Depending  on  the  scenario  and  the  information  that  is  under  scrutiny,  it  may  be 
difficult  to  discern  what  experience  actually  entails.  For  instance,  experience  may  consist 
in  part  of  remembered  information  that  can  be  directly  compared  to  information  that  is  of 
interest.  A  memorized  fact  may  be  compared  to  a  statement  or  message  about  the  fact  (or 
an  erroneous  version  of  it)  to  determine  if  the  message  matches  the  memory.  It  is  also 
possible  that  information,  for  which  a  person  has  no  memory,  will  be  compared  to 
memories  of  other  similar  information.  For  instance,  expectations  about  the 
characteristics  of  the  new  information  may  be  compared  to  the  actual  characteristics  of 
that  information;  any  discrepancies  might  violate  maxims  of  the  Grice’s  Cooperative 
Principle  or  IMT  (Grice,  1975;  McCornack,  1992). 

Conceptual  Model 

The  concepts  and  hypothesized  associations  that  are  of  interest  in  this  study  were 
compiled  into  a  graphical  view  as  shown  in  Figure  2.  A  more  detailed  explanation  of  the 
model  and  the  hypotheses  that  it  depicts  is  presented  below. 
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Figure  2.  Conceptual  Model  for  Association  Hypotheses  of  Familiarity, 
Relevance,  Definite  Answers,  Accurate  Answers. 


Model  Development 

Despite  the  extant  research  that  has  already  been  carried  out  and  the  knowledge 
that  has  been  gained  thus  far  about  the  constructs  that  are  presented  above,  more  research 
is  needed  in  this  area.  In  an  effort  to  perform  research  in  an  area  that  has  not  been 
heavily  covered,  associations  between  constructs  that  have  not  been  previously  associated 
were  examined.  The  constructs  of  relevance,  familiarity,  and  accuracy  do  not  appear  to 
have  been  previously  associated,  or  tested. 

Conceptually,  it  may  be  possible  for  a  task  of  some  kind  to  cause  certain 
information  to  be  relevant  to  a  person  who  must  perform  the  task.  It  may  further  be 
supposed  that  the  person  might  seek  the  information  that  is  relevant  to  him,  and  that  he 
might  become  familiar  with  the  infonnation  in  doing  so.  A  person  who  is  familiar  with 
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the  information,  and  to  whom  this  information  is  relevant  might  be  considered  a  subject 
matter  expect.  A  subject  matter  expert  might  in  turn  be  equipped  to  evaluate  relevant 
information,  and  to  perform  a  task  such  as  determine  whether  the  infonnation  has  been 
manipulated  or  not.  This  subject  matter  expert,  if  confident  in  such  a  determination, 
might  be  inclined  to  state  their  in  a  definite  sense. 

Under  the  above  scenario,  the  constructs  of  relevance,  familiarity,  willingness  to 
commit  to  definite  answers,  and  judgment  accuracy  of  whether  infonnation  has  been 
manipulated  might  be  interrelated.  If  any  of  the  set  of  possible  associations  were 
significantly  supported,  then  further  research  could  be  perfonned  in  order  to  determine 
what  portions  of  those  constructs  influence  performance  of  text  manipulation 
determinations. 

Hypotheses 

With  these  questions,  a  set  of  hypotheses  was  developed  about  the  relationship 
between  each  of  these  concepts.  To  represent  a  potential  association  between  each  one, 
six  hypotheses  were  developed.  These  hypotheses  are: 

Hypothesis  1:  Familiarity  positively  associates  with  Accurate  Answers.  This 
hypothesis  aims  to  detennine  whether  the  concept  of  overall  domain  experience  (Biros,  et 
al,  2002;  Klein,  et  al,  1997b)  with  the  topic  of  the  text  information  associates  with 
improved  accuracy  in  detecting  whether  manipulation  is  present. 

Hypothesis  2:  Definite  Answers  positively  associate  with  Accurate  Answers. 
This  hypothesis  aims  to  detennine  the  participant  is  confident  in  their  detennination  of 
whether  manipulation  is  present  in  the  information  that  is  presented,  or  more  specifically 
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that  the  person  felt  that  they  were  up  to  the  task  of  detennining  whether  a  particular  set  of 
information  was  manipulated  (Harrison,  et  al,  1997). 

Hypothesis  3:  Relevance  positively  associates  with  Accurate  Answers.  This 
hypothesis  aims  to  determine  whether  the  concept  of  information  relevance,  as  described 
by  McCornack  (1992)  associates  with  improved  accuracy  in  detecting  whether 
manipulation  is  present. 

Hypothesis  4:  Familiarity  positively  associates  with  definite  answers.  This 
hypothesis  seeks  to  determine  whether  there  is  an  association  between  familiarity  as 
described  above  in  Hypotheses  1  and  participant  willingness  to  commit  to  definite 
answers  as  described  above  in  Hypothesis  2. 

Hypothesis  5:  Relevance  positively  associates  with  definite  answers.  This 
hypothesis  aims  to  determine  whether  there  is  an  association  between  relevance  and 
participant  willingness  to  commit  to  a  definite  answer,  as  described  above  in  Hypotheses 
2  and  3. 

Hypothesis  6:  Relevance  positively  associates  with  Familiarity.  Lastly,  this 
hypothesis  aims  to  determine  whether  the  concept  of  whether  the  infonnation  relevance, 
as  described  by  McCornack  (1992)  associates  with  the  concept  of  overall  domain 
experience  (Biros,  et  al,  2002;  Klein,  et  al,  1997b)  with  the  topic  of  the  text  information. 
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Chapter  3  -  Methodology 


Overview  of  the  Method 

The  study  is  quantitative  in  nature,  and  utilizes  a  combination  of  pre-experimental 
and  correlational  design  characteristics  (Leedy  and  Ormrod,  2001)  where  a  group  of 
constructs  was  measured  in  order  to  determine  their  associations  with  each  other.  This 
study  utilized  an  experiment,  a  pool  of  Air  Force  Institute  of  Technology  (AFIT) 
students,  and  an  information  pool  that  was  drawn  from  a  weekly  informational  E-mail 
that  is  distributed  to  them.  The  group  was  exposed  to  one  treatment  (a  set  of  information 
paragraphs,  some  of  which  were  manipulated  and  some  that  were  not)  that  was  the  same 
for  all  participants  and  one  observation  (where  participants  were  asked  for  their 
perceptions  about  all  of  the  text  paragraphs). 

Experimental  development 

In  view  of  McCornack’s  (1992)  study  of  IMT,  with  its  focus  on  the  content  and 
meaning  of  infonnation,  and  Biros,  et  al’s  (2002)  study  of  manipulated  data  in  an 
information  system  where  system  experts  were  asked  to  detect  information  problems,  an 
experiment  was  envisioned  to  examine  deception  in  text-based  scenarios.  It  was 
envisioned  that  authentic  text  messages  might  be  manipulated,  and  that  a  pool  of 
participants  could  be  asked  to  determine  whether  manipulation  was  present.  It  was  also 
conceived  that  the  pool  of  information  could  be  drawn  from  information  that  already 
exists  and  that  participants  have  dealt  with  in  some  fashion  already. 

The  answers  that  the  participants  gave  about  their  perceptions  of  the  information 
were  analyzed  to  detennine  whether  there  was  an  association  between  some 
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characteristics  that  the  participants  perceived  about  each  infonnation  paragraph.  For  each 
paragraph,  participants  were  asked  how  relevant  they  perceived  the  topic  to  be  to  them. 
They  were  also  asked  how  familiar  overall  they  were  with  the  topic.  Finally,  the 
participants  were  asked  for  their  perception  about  whether  a  paragraph  had  been 
manipulated  or  not  (for  two  parts  of  the  battery,  the  participants  were  given  the  choice  of 
yes  or  no,  for  the  third  part,  participants  were  asked  which  one  of  two  paragraphs  was  the 
manipulated  one).  As  part  of  the  manipulation  answer,  the  participants  were  able  to 
answer  with  a  more  confident  sense  by  selecting  an  answer  that  contained  the  word 
“Definitely.”  The  data  collected  was  analyzed  to  determine  which  items  significantly 
associated  with  one  another. 

To  test  the  six  hypotheses  that  were  developed  for  this  study,  an  analysis  of  the 
variance  of  the  collected  data  at  an  alpha  of  0.05  was  used.  An  alpha  of  0.05  is 
commonly  used  in  management  information  systems  research. 

Description  of  the  Experiment 

Participants  were  given  a  fictitious,  yet  plausible  scenario  to  set  the  stage  for  the 
questionnaire.  This  scenario  explained  that  a  person  that  they  have  worked  with  was 
researching  a  set  of  information  items  as  part  of  an  important  and  time-critical  task,  and 
that  part  of  this  task  was  to  determine  whether  the  available  infonnation  had  been 
manipulated  or  not.  It  explained  that  the  information  (which  the  participants  have  been 
exposed  to  on  a  weekly  basis  prior  to  this  experiment)  was  retrieved  from  an  information 
archive  with  an  unknown  level  of  security  and  that  alternate  sources  of  that  information 
were  unavailable  at  the  moment. 


23 


Participants  were  exposed  to  twenty-four  information  paragraphs  that  were 
presented  in  three  different  formats  (eight  paragraphs  in  each  format):  standalone,  quota, 
and  comparison.  In  each  of  three  sections  of  the  experiment,  a  sub-scenario  provided 
additional  information  that  was  germane  to  the  instructions  for  each  part.  Information 
that  was  given  in  each  sub-scenario  was  only  applicable  within  its  respective  section,  as 
described  below. 

In  the  standalone  format,  the  participants  were  asked  to  evaluate  each  of  the  eight 
text  paragraphs  on  an  individual  basis  and  to  make  a  determination  as  to  whether  the  item 
had  been  manipulated  or  not.  They  were  told  that  it  was  possible  that  all,  some  or  none 
of  the  items  were  manipulated;  in  actuality,  four  of  the  eight  were  manipulated. 

In  the  quota  format,  the  participants  were  provided  new  additional  information  in 
the  scenario  that  told  them  that  for  this  next  set  of  eight  messages,  four  of  them  had 
definitely  been  manipulated,  but  that  it  was  unknown  which  of  the  eight  paragraphs  those 
were.  This  scenario  information  was  true  in  that  four  of  the  eight  messages  in  this  section 
were  actually  manipulated. 

In  the  last  format,  comparison,  participants  were  shown  two  versions  of  a  given 
information  paragraph,  the  original  (unmodified)  and  a  manipulated  alternative.  The  sub¬ 
scenario  specified  that  for  the  eight  messages  in  this  part,  the  archive  system  drew  the 
information  paragraphs  from  two  source  systems  and  that  for  the  purposes  of  this 
scenario:  one  of  the  paragraphs  came  from  a  secure  system  and  the  other  was 
compromised,  but  that  no  indication  was  given  as  to  which  version  came  from  which 
system.  The  participants  were  asked  to  examine  the  infonnation  and  to  choose  which  of 
the  items  was  manipulated. 
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The  participants  were  asked  to  determine  whether  or  not  they  feel  the  infonnation 
has  been  manipulated,  in  what  manner  that  it  might  have  been  changed,  and  to  give  their 
perception  of  other  characteristics  of  the  information  as  well  (whether  they  feel  that  they 
were  familiar  with  the  information,  and  whether  the  information  was  somehow  relevant 
to  them).  The  participants  also  were  encouraged  to  make  comments  on  what  factors  they 
considered  in  making  their  detennination. 

The  volunteers  were  sought  among  students  of  the  Air  Force  Institute  of 
Technology  for  purposes  of  (1)  convenience  (access  to  persons),  (2)  similarity  of  these 
persons  to  future  Air  Force  beneficiaries  of  the  new  knowledge  (the  research  is  carried 
out  on  behalf  of  the  Air  Force  Office  of  Scientific  Research  to  gain  insight  into  deception 
that  may  occur  in  a  military  organization),  and  (3)  to  utilize  an  environment  where  there 
exists  an  existing  textual  information  pool  that  is  familiar  to  the  participants,  and  that  can 
be  drawn  upon  for  use  in  the  experiment. 

The  Existing  Information  Pool 

The  information  that  is  used  in  this  experiment  consists  of  excerpts  that  are  drawn 
from  a  weekly  informational  message  (“What  Is  Going  On”  or  “WIGO”)  that  is  issued 
via  E-mail  from  the  Aeronautical  Systems  Center  Commander,  United  States  Air  Force 
Lieutenant  General  Richard  V.  Reynolds.  The  Aeronautical  Systems  Center  is  a  large 
organization  located  at  Wright  Patterson  Air  Force  Base,  Ohio,  which  has  close  ties  to  the 
Air  Force  Institute  of  Technology  (AFIT)  due  to  its  location  and  on-going  cooperative 
research  efforts.  The  AFIT  Commandant  or  Vice  Commandant  forwards  the  WIGO  to 
the  faculty,  staff  and  students  of  AFIT. 
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In  WIGO  messages  published  between  13  May  2002  and  13  January  2003,  there 
were  approximately  357  informational  paragraphs  that  covered  a  variety  of  current  event 
topics  that  are  generally  of  interest  to  personnel  assigned  to  Wright  Patterson  Air  Force 
Base,  the  Aeronautical  Systems  Center,  and  the  Air  Force  Institute  of  Technology 
(AFIT).  While  inspecting  the  topics  for  suitability  for  the  experiment,  it  was  noticed  that 
individual  topics  vary  widely  in  their  potential  relevance  to  the  experiment  participants 
(AFIT  personnel);  for  example,  some  items  were  narrowly  specific  to  a  small  portion  of 
the  Aeronautical  Systems  Center.  Other  topics  were  clearly  of  a  base-wide  or  Air  Force 
wide  interest. 

Since  the  scope  of  this  experiment  involves  the  measurement  of  familiarity  and 
relevance  of  information  to  the  participants  in  addition  to  their  accuracy  at  detecting 
manipulation,  items  were  selected  from  the  entire  infonnation  set  that  would  likely 
provide  some  degree  of  familiarity  and  relevance  to  the  target  audience  of  AFIT  students. 

Initially,  approximately  96  items  were  chosen  by  the  author  of  this  experiment 
from  the  357  available  informational  paragraphs.  These  were  chosen  by  the  author 
subjectively  with  the  rationale  that  the  topic  was  likely  to  be  of  some  level  of  interest  to 
the  AFIT  student  participant  pool.  Later,  the  96  candidate  messages  were  narrowed  to 
the  24  that  were  actually  used  in  the  experiment. 

Each  item  was  examined  for  suitability  for  manipulation  in  accordance  with  the 
four  types  that  were  selected  for  use  in  the  experiment.  In  order  to  be  considered  a 
candidate  for  a  test  item,  the  infonnation  paragraph  had  to  have  potential  to  be  changed  in 
a  way  that  would  significantly  and  plausibly  alter  its  meaning  to  some  reasonable  portion 
of  the  participants.  A  suitable  manipulation  is  one  that  changes  the  meaning  of  the 
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information  item  significantly  so  that  there  is  some  likelihood  that  a  participant  will 
accept  the  item  as  true.  A  truly  deceptive  act  has  an  objective  and  in  this  case,  it  is  to 
alter  a  participant’s  perception  of  a  topic  or  an  information  item. 

Since  the  WIGO  messages  are  informational,  a  successful  deception  would 
misinform  the  target  population  about  the  topic.  Since  little  work  has  been  published  on 
detecting  deception  in  this  area,  the  approach  to  manipulating  the  infonnation  comprised 
a  change  that  significantly  affected  the  meaning  of  the  message  in  one  of  four  ways. 

The  WIGO  information  items  are  real  world  and  vary  in  terms  of  what  is  covered 
and  how  this  information  is  expressed,  (i.e.  some  lend  themselves  strictly  to  a  verbal 
description,  other  present  some  numerical  value  that  serves  as  the  essence  of  the 
message).  As  outlined  in  Chapter  2,  four  specific  methods  of  manipulation  were 
employed:  significant  value-change,  order  of  magnitude  value-change,  logic  alteration, 
and  reattribution.  Objectives  for  each  type  are  explained  in  reference  to  the  experiment 
below. 

Manipulation  Types 

A  significant  quantitative  manipulation  is  one  that  changes  a  descriptive  number 
within  the  message  in  such  a  way  that  it  changes  the  meaning  of  the  item.  For  purposes 
here,  this  generally  was  restricted  to  a  doubling  or  a  halving  of  a  value.  For  example,  if  a 
WIGO  message  announced  that  there  are  “six  months  left”  to  complete  a  professional 
development  training  course,  the  manipulation  might  change  the  statement  to  instead  say 
that  there  are  “twelve  months  left”.  In  this  scenario,  a  successful  deception  would  lead 
readers  to  believe  they  had  more  time  to  complete  the  course  than  they  really  did.  If  the 
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deceiver’s  goal  was  to  make  people  late  for  completion  of  the  training,  those  who 
believed  that  they  had  twelve  months  left  and  waited  to  do  so  until  after  the  real  deadline 
would  be  late;  and  in  those  cases  the  deceiver  would  be  successful.  This  category 
appeared  to  be  suitable  for  some  of  the  information  items,  however  it  was  seen  that  some 
of  the  information  items  did  not  change  their  meaning  with  a  mere  50%,  or  200% 
manipulation. 

In  some  cases  where  the  50%  or  200%  manipulation  did  not  seem  to  be 
significant  enough,  an  order  of  magnitude  quantitative  manipulation  was  applied  in  order 
to  create  an  effective  manipulation.  Multiplying  or  dividing  a  number  by  10,  or  possibly 
100,  affected  the  meaning  of  the  statement  when  a  50%  or  200%  change  did  not. 

In  other  cases,  a  numerical  change  was  not  as  appropriate  or  effective  as  a  change 
in  logic.  In  general,  the  method  used  was  to  give  the  message  an  opposite  meaning.  For 
instance  if  an  announcement  described  something  as  a  success,  an  effective  manipulation 
might  say  that  the  effort  was  a  failure.  In  some  instances,  the  message  contained 
supporting  or  explanatory  infonnation  that  was  in  addition  to  the  supporting  statement; 
these  supporting  statements  were  deleted,  changed,  or  replaced  as  needed  to  support  the 
core  logic  change. 

The  last  type  of  manipulation  was  one  of  reattribution.  When  a  person  or 
organization  was  credited  with  something,  it  was  sometimes  plausible  to  change  names 
and  reattribute  the  actions  themselves.  For  instance,  one  item  gave  news  of  US  Army 
personnel  being  sent  to  Wright  Patterson  Air  Force  Base  to  augment  Air  Force  Security 
Forces  personnel;  a  reattribution  in  this  case  might  say  that  the  Air  Force  sent  some  of  its 
personnel  to  an  Anny  site  instead. 
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Each  of  the  24  messages  that  were  used  was  manipulable  via  at  least  one  of  the 
manipulation  formats.  Some  of  the  messages  lent  themselves  to  manipulation  in  more 
than  one  of  the  ways,  however  only  one  is  required  for  use  in  the  experiment.  This 
requirement  controlled  for  possible  errors  that  would  result  from  messages  that  inherently 
would  have  little  plausibility  in  it  were  to  be  altered  in  one  of  the  four  chosen  manners. 

Experiment  Scenario 

During  the  experiment,  accurate  and  manipulated  messages  in  three  different 
formats  were  presented.  For  all  scenarios,  it  was  explained,  “An  AFIT  staff  member  who 
is  a  close  friend  of  yours  has  been  tasked  to  research  a  number  of  items  that  were  covered 
in  WIGO  messages  over  the  past  few  months  on  a  short  deadline.  Unfortunately,  the  task 
was  received  late  in  the  day  and  the  results  are  due  first  thing  in  the  morning.  Some 
problems  with  the  base  network  have  limited  the  research  sources  that  are  available  now 
to  one,  an  archive  that  contains  past  WIGO  messages.  In  addition,  your  friend  tells  you 
that  this  particular  system  does  not  appear  to  have  the  latest  security  patches  applied  to  it 
and  that  the  integrity  of  the  information  might  be  in  some  doubt.  You  agree  to  help.” 

“To  start  with,  eight  infonnation  items  are  called  up  in  the  system.  Determine  if 
each  of  these  items  is  accurate  or  whether  they  have  been  changed  in  some  way.  To  give 
some  idea  of  your  confidence  in  your  assessment,  please  answer  the  following  additional 
questions  about  this  item.” 

In  the  first  battery,  eight  messages  were  shown  with  questions  about  each  passage. 
Half  of  these  will  be  accurate  and  half  will  be  manipulated.  Next  to  each  passage,  the 
participants  will  be  asked  whether  they  are  “Not  Familiar,”  “Minimally  Familiar,” 
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“Somewhat  Familiar,”  “Familiar,”  or  “Quite  Familiar.”  with  the  message  topic.  The 
second  question  they  were  asked  is  whether  they  consider  this  information  to  be  “Not 
Relevant,”  “Minimally  Relevant,”  “Somewhat  Relevant,”  “Relevant,”  or  “Clearly 
Relevant”  to  them  on  some  level.  Most  importantly,  the  participants  were  asked  to 
determine  if  there  is  manipulation  present  in  the  message  in  the  following  format: 
“Definitely  Not,”  “Probably  Not,”  “Probably  Yes,”  and  “Definitely  Yes.” 

After  the  first  battery  of  questions  was  completed,  a  quota  of  four  manipulated 
information  passages  was  introduced  for  the  next  set  of  eight.  The  scenario  explained,  “a 
partially  implemented  feature  of  the  archive  is  familiar  to  your  friend.  An  indicator 
shows  that  there  were  recent  edits  to  four  of  the  eight  information  items  that  she  needs, 
but  does  not  identify  which  items  have  been  edited.  This  is  very  unusual  because  this  is 
an  archive  and  the  WIGO  items  should  not  be  changed  once  they  are  in  the  database. 
Since  all  eight  are  of  vital  importance  to  her,  she  needs  your  help  to  assess  which  four  of 
the  items  were  manipulated.” 

In  the  second  battery  of  messages,  as  stated  above  a  quota  approach  was  used. 
Participants  were  shown  a  set  of  messages  and  were  asked  questions  similar  to  those  in 
the  first  battery.  The  goal  here  was  the  engage  the  participant  to  select  a  known  number 
of  manipulated  messages  and  to  help  control  for  a  bias  where  participants  might  suspect 
all  of  the  messages  or  none  of  them. 

In  the  last  battery,  the  participants  were  told,  “while  accessing  the  last  eight  items 
that  are  needed,  she  noticed  that  there  were  duplicate  entries  for  those  particular  items. 
Using  another  partially  implemented  software  feature,  comparisons  were  made  between 
the  duplicate  entries  and  in  each  case  the  duplicates  differed  from  one  another — in  other 
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words,  you  have  two  versions  of  each  item,  and  those  versions  conflict  with  other.  Since 
the  feature  is  only  partially  implemented,  it  does  not  provide  any  other  information,  but 
she  assumes  that  one  message  in  each  pairing  must  be  correct.  Which  one  is  correct?” 

In  the  last  battery,  participants  were  shown  the  correct  and  a  manipulated  version 
of  the  same  message  side  by  side — the  portions  of  the  text  that  differed  from  each  other 
were  highlighted  so  that  the  participants  could  focus  on  differentiating  the  items  and 
selecting  a  correct  one.  They  were  asked  to  select  which  version  contains  the 
manipulation,  and  were  again  asked  the  questions  about  familiarity  and  relevance  per 
above.  Different  from  the  first  and  second  batteries  is  the  selection  statement  for  their 
choice.  The  choices  will  be  phrased  “Definitely  Message  A,”  “Probably  Message  A,” 
Probably  Message  B,”  and  “Definitely  Message  B.” 

Participant  Pool 

Volunteers  were  invited  to  participate  via  an  E-mail  message  that  was  sent  to  all 
Air  Force  Institute  of  Technology  graduate  students  who  are  enrolled  in  one  of  the 
Engineering  or  Management  programs  at  the  AFIT.  The  advertisement  stated  that  the 
request  was  for  volunteers  to  participate  in  an  experiment  that  involved  text 
manipulation.  Participants  were  met  as  their  schedule  permitted  to  perform  the 
experiment,  typically  in  a  classroom  or  library  setting,  and  most  often  in  small  groups  of 
four  to  six  people,  and  they  used  approximately  forty  to  sixty  minutes  to  complete  the 
instrument. 

Answer  sheets  were  checked  for  completeness  to  the  extent  possible  as  they  were 
turned  in,  however  a  small  number  of  answers  were  later  found  to  be  missing  despite  the 
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effort.  The  data  that  the  participants  provided  was  entered  into  a  Microsoft  Excel 
2000/XP  spreadsheet,  and  was  imported  to  SAS  Institute’s  JMP-IN  version  4  statistical 
analysis  software. 

Execution  of  the  Experiment 

Thirty-seven  volunteer  participants  took  part  in  the  experiment.  These  persons 
were  met  at  their  convenience  in  classroom  and  library  settings.  The  time  required  to 
complete  the  instrument  was  between  forty  and  sixty  minutes  in  all  cases. 

The  criterion  for  an  accurate  answer  depended  upon  the  format  in  which  the  text 
excerpt  was  presented.  In  the  first  two  sections,  a  manipulated  message  needed  to  be 
identified  as  “Yes... manipulated”  and  a  non-manipulated  message  needed  to  be  identified 
as  “Not... manipulated.”  In  the  comparative  section  where  the  participant  was  given  two 
passages  to  choose  from,  the  participant  had  to  correctly  identify  which  of  the  two  was 
manipulated,  either  Message  A  or  Message  B.  There  was  only  one  correct  answer  for 
each  question. 

A  small  number  of  answers  were  not  completed  by  the  participants;  these  blanks 
appeared  to  be  distributed  in  no  particular  manner  among  manipulation,  familiarity,  and 
relevance  answers.  Since  the  number  of  these  blanks  was  small,  and  no  obvious 
explanation  was  apparent  (i.e.  they  did  not  appear  to  be  associated  with  any  particular 
questions,  for  instance)  the  blanks  were  ignored  automatically  in  the  statistical  tests 
performed  by  JMP-IN. 
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Chapter  4  -  Analysis  and  Results 


Descriptive  Statistics  of  the  Data 

Shown  below  is  a  chart  of  the  accuracy  rates  of  the  experiment.  The  group  of 
participants  as  a  whole  accurately  identified  which  messages  were  manipulated,  and 
which  were  not,  approximately  62  percent  of  the  time  overall  (bottom  right  corner  of  the 
chart  below.  Figure  3  contains  a  summary  of  the  rates  of  correct  answers. 


Figure  3.  Accuracy  Rates  Overall  and  Broken  Down  by  Presentation  Format 
and  Manipulation  Type 


Format 

Manipulation  Type 

Value 

Significant 

Value 

Magnitude 

Logic 

Change 

Identity 

Change 

None 

All 

Types 

Stand  Alone 

0.38 

0.22 

0.81 

0.36 

0.66 

0.55 

Quota 

0.35 

0.70 

0.70 

0.54 

0.60 

0.59 

Compare 

0.38 

0.78 

0.73 

0.96 

0.71 

All  Formats 

0.37 

0.62 

0.74 

0.71 

0.63 

0.62 

Accuracy  for  each  question  individually  ranged  from  22  to  97  percent.  It  is 
interesting  to  note  that  when  a  given  message  contained  no  manipulation  whatsoever,  the 
non-manipulated  information  was  properly  identified  as  non-manipulated  only  63  percent 
of  the  time.  It  is  also  interesting  to  note  that  the  results  for  the  standalone  and  quota 
formats,  the  results  are  not  much  better  than  chance.  The  results  for  specific  questions 
can  be  reviewed  in  the  chart  in  Figure  4. 
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Figure  4.  Summary  of  Questions,  and  Rates  of  Definite  Answers  and  Accuracy 
For  Each  Question. 


Standalone:  No  Quota 


Question 

A1 

A2 

A3 

A4 

A5 

A6 

A7 

A8 

Manipulated? 

Yes 

No 

No 

Yes 

Yes 

No 

No 

Yes 

Definite  Answer 

.24 

.19 

.14 

.49 

.25 

.27 

.35 

.24 

Correct  Answer 

.22 

.81 

.62 

.81 

.36 

.59 

.59 

.38 

Quota:  4  of  8  are  known  to  be  manipulated 


Question 

B1 

B2 

B3 

B4 

B5 

B6 

B7 

B8 

Manipulated? 

No 

Yes 

Yes 

No 

No 

Yes 

Yes 

No 

Definite  Answer 

.38 

.35 

.49 

.30 

.35 

.35 

.38 

.19 

Correct  Answer 

.27 

.70 

.54 

.86 

.76 

.35 

.70 

.51 

Comparative:  1  is  manipulated,  identify  it 


Question 

Cl 

C2 

C3 

C4 

C5 

C6 

C7 

C8 

Definite  Answer 

.81 

.32 

.28 

.92 

.41 

.11 

.65 

.89 

Correct  Answer 

.97 

.73 

.83 

.95 

.62 

.40 

.35 

.84 

The  accuracy  rate  varied  widely  among  the  questions.  It  must  be  acknowledged 
that  each  question  was  unique  in  tenns  of  the  topic  it  covered,  the  specific  type  of 
information  it  contained,  whether  it  was  presented  with  or  without  manipulation,  what 
type  of  manipulation  it  was  subjected  to,  how  the  manipulation  was  applied,  and  in  what 
format  the  question  was  presented  to  the  participants.  These  factors  may  have  influenced 
the  accuracy  rate  in  ways  that  cannot  be  accounted  for  by  the  analysis  performed  within 
this  thesis.  Despite  the  wide  range  of  accuracy  rates  for  the  questions,  it  is  still 
reasonable  to  assume  that  the  associations  that  are  posited  in  the  hypotheses  can  be 
examined. 

As  stated  previously  all  of  the  hypotheses  were  tested  using  a  one-way  analysis  of 
variance  (ANOVA)  with  an  alpha  of  0.05;  0.05  is  a  commonly  used  level  of  alpha  in 


34 


management  information  system  research.  The  results  and  observations  for  each 
hypothesis  follow. 

HI:  Familiarity  Associates  with  Accurate  Answers 

This  hypothesis  suggests  that  rates  of  accuracy  will  vary  significantly  with 
participant  familiarity  with  the  topic.  The  ANOVA  was  run  on  870  matched  pairs  of 
answers  for  familiarity  and  manipulation  judgment  accuracy.  The  ANOVA  gave  a  result 
of  F(4,865)  =  1.15;  p  =  0.3302.  This  does  not  support  Hypothesis  1. 

Visual  inspection  of  the  JMP-IN  version  4  grouped  means  and  variance  chart 
showed  a  nearly  flat-line  response  with  mean  accuracy  rates  near  0.60,  with  a  small,  but 
visible  upward  trend.  Examination  of  the  95%  confidence  intervals  for  each  of  the  five 
levels  of  familiarity  showed  a  wide  range  for  likely  accuracy  rates.  A  Tukey-Kramer 
analysis  revealed  no  matched  pairs  with  significant  differences  between  them. 

Hypothesis  1  is  not  supported.  The  JMP-IN  4  output  page  for  an  analysis  of 
variance  of  these  constructs  is  displayed  in  Figure  5. 
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Figure  5.  JMP  Output:  One-Way  Analysis  of  Manipulation  Judgment  Accuracy 
By  Familiarity  (Accuracy  Rates  for  Each  Degree  of  Familiarity) 


Oneway  Anova 
Summary  of  Fit 
Rsquare  0.005304 

Adj  Rsquare  0.000705 

Root  Mean  Square  Error  0.486444 
Mean  of  Response  0.616092 
Observations  (or  Sum  Wgts)  870 


Source  DF 
Familiarity  4 
Error  865 
C.  Total  869 


Analysis  of  Variance 

Sum  of  Squares  Mean  Square  F  Ratio  Prob  >  F 
1.09151  0.272878  1.1532  0.3302 

204.68320  0.236628 

205.77471 


Means  for  Oneway  Anova 


Level  Number 

Mean 

Std  Error  Lower  95% 

Upper  95% 

1 

242 

0.582645 

0.03127 

0.52127 

0.64402 

2 

182 

0.637363 

0.03606 

0.56659 

0.70813 

3 

187 

0.582888 

0.03557 

0.51307 

0.65271 

4 

194 

0.644330 

0.03492 

0.57578 

0.71288 

5 

65 

0.692308 

0.06034 

0.57389 

0.81073 

Std  Error  uses  a  pooled  estimate  of  error  variance 
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H2:  Definite  Answers  associate  with  Accurate  Answers 

Hypothesis  2  suggests  that  the  rate  of  accuracy  will  associate  with  definite 
answers.  The  result  of  the  ANOVA  was  F(l,882)  =  11.03;  p  =  0.0009,  which  supports 
Hypothesis  2. 

Visual  inspection  of  the  grouped  means  revealed  a  clearly  increased  mean 
accuracy  in  the  definite  answer  group  over  the  indefinite  answer  group.  Examination  of 
the  95%  confidence  intervals  for  those  with  and  without  definite  answers  revealed  a  pair 
of  ranges  that  did  not  overlap. 

Hypothesis  2  is  supported.  The  JMP-IN  4  output  page  for  an  analysis  of  variance 
of  these  constructs  is  displayed  in  Figure  6. 
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Figure  6.  JMP  Output:  One-Way  Analysis  of  Manipulation  Judgment  Accuracy 
By  Definite  Answers  (Accuracy  Rates  for  Definite  Answers  versus  non-Definite 
Answers) 


Oneway  Analysis  of  Accurate  By  Definite 

1  “I  - 

0.9- 
0.8- 
0.7- 
3  0.6- 

ro 

§  0.5- 
<  0.4- 
0.3- 
0.2- 
0.1- 

0^ _ - _ _ ■ 

0  1 

Definite 


Oneway  Anova 
Summary  of  Fit 


Rsquare  0.012351 

Adj  Rsquare  0.011231 

Root  Mean  Square  Error  0.48377 

Mean  of  Response  0.616516 

Observations  (or  Sum  Wgts)  884 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square  F  Ratio 

Prob  >  F 

Definite 

1 

2.58126 

2.58126  11.0295 

0.0009 

Error 

882 

206.41761 

0.23403 

C.  Total 

883 

208.99887 

Means  for  Oneway  Anova 

Level 

Number 

Mean 

Std  Error  Lower  95%  Upper  95% 

0 

539 

0.573284 

0.02084  0.53239 

0.61418 

1 

345 

0.684058 

0.02605  0.63294 

0.73518 

Std  Error  uses  a  pooled  estimate  of  error  variance 


38 


H3:  Relevance  associates  with  Accurate  Answers 


This  hypothesis  suggests  that  rates  of  accuracy  will  differ  depending  on  the 
message’s  relevance  to  the  participant.  The  ANOVA  results  were  F(4,873)  =  1.5152; 
p  =  0.1957.  Since  this  exceeds  our  alpha  of  0.05,  Hypothesis  3  is  not  supported. 

A  Tukey-Kramer  analysis  was  also  run  on  these  items,  and  again  no  statistical 
support  for  the  hypothesis  was  found  with  any  pairing.  Visually,  accuracy  based  on  the 
relevance  answers  seemed  to  be  nearly  flat  with  an  inverse  trend. 

Hypothesis  3  is  not  supported.  The  JMP-IN  4  output  page  for  an  analysis  of 
variance  of  these  constructs  is  displayed  in  Figure  7. 
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Figure  7.  JMP  Output:  One-Way  Analysis  of  Manipulation  Judgment  Accuracy 
By  Relevance  (Accuracy  Rates  for  Each  Degree  of  Relevance) 


Oneway  Analysisof  Accurate  By  Relevance 


1- 

0.9- 

0.8- 

0.7- 

£  0.6- 
03 

3  0.5- 
<  0.4- 

0.3- 

0.2- 

0.1- 

0- 

■ 

1 

2 

i 

3  4  5 

Relevance 

All  Pairs 

Tukey-Kramer 

0.05 

Oneway  Anova 
Summary  of  Fit 


Rsquare  0.006895 

Adj  Rsquare  0.002344 

Root  Mean  Square  Error  0.486294 

Mean  of  Response  0.615034 

Observations  (or  Sum  Wgts)  878 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Prob  >  F 

Relevance 

4 

1.43327 

0.358317 

1.5152 

0.1957 

Error 

873 

206.44828 

0.236481 

C.  Total 

877 

207.88155 

Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

1 

107 

0.682243 

0.04701 

0.58997 

0.77451 

2 

133 

0.676692 

0.04217 

0.59393 

0.75945 

3 

214 

0.579439 

0.03324 

0.51420 

0.64468 

4 

251 

0.589641 

0.03069 

0.52940 

0.64989 

5 

173 

0.606936 

0.03697 

0.53437 

0.67950 

Std  Error  uses  a 

pooled  estimate 

of  error  variance 
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H4:  Familiarity  associates  with  Definite  Answers 

This  hypothesis  posits  that  familiarity  associates  with  definite  answers. 
Intuitively,  it  is  reasonable  to  expect  that  if  an  association  does  exist,  that  it  might  be  a 
directly  proportional  relationship.  The  result  for  the  ANOVA  was  F(4,865)  =  20.54; 
p<  0.0001.  This  extremely  low  p-value  strongly  supports  that  there  is  an  association 
between  familiarity  and  definite  answers. 

The  Tukey-Kramer  analysis  was  also  run  on  this  data,  and  it  too  showed  strong 
support  for  this  hypothesis.  Most  of  the  familiarity  ratings  showed  significantly  different 
definite  answer  rates,  particularly  in  the  higher  end  of  the  familiarity  scale.  Visually,  all 
the  ratings  appeared  to  have  a  proportional  and  direct  association. 

Hypothesis  4  is  supported.  The  JMP-IN  4  output  page  for  an  analysis  of  variance 
of  these  constructs  is  displayed  in  Figure  8. 
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Figure  8.  JMP  Output:  One-Way  Analysis  of  Definite  Answer  Rate  By 
Familiarity  (Definite  Answer  Rates  for  Each  Degree  of  Familiarity) 


Oneway  Analysis  of  Definite  By  Familiarity 


Oneway  Anova 
Summary  of  Fit 
Rsquare  0.086726 

Adj  Rsquare  0.082503 

Root  Mean  Square  Error  0.46714 

Mean  of  Response  0.388506 

Observations  (or  Sum  Wgts)  870 


Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 
Familiarity  4  17.92501  4.48125  20.5355 

Error  865  188.76005  0.21822 

C.  Total  869  206.68506 


Prob  >  F 
<.0001 


Means  for  Oneway  Anova 


Level  Number 

Mean 

Std  Error  Lower  95% 

1 

242 

0.243802 

0.03003 

0.18486 

2 

182 

0.302198 

0.03463 

0.23424 

3 

187 

0.379679 

0.03416 

0.31263 

4 

194 

0.556701 

0.03354 

0.49087 

5 

65 

0.692308 

0.05794 

0.57859 

Upper  95% 
0.30274 
0.37016 
0.44673 
0.62253 
0.80603 


Std  Error  uses  a  pooled  estimate  of  error  variance 
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H5:  Relevance  associates  with  Definite  Answers 

This  hypothesis  is  similar  to  hypothesis  4,  however  it  compares  relevance  to  the 
rate  of  definite  answers  instead  of  familiarity.  The  ANOVA  results  were:  F(4,873)  = 
9.6363;  p  <  0.0001,  which  strongly  supports  that  a  significant  association  exists  between 
relevance  and  the  rate  of  definite  answers.  A  Tukey-Analysis  revealed  statistically 
significant  association  between  different  ratings  for  relevance,  but  did  not  show 
significant  support  for  definite  answer  rate  difference  between  all  pairings  of  relevance 
ratings.  Visually,  the  association  does  not  appear  to  be  as  dramatic,  or  as  smooth  as  the 
familiarity  and  definite  answer  relationship. 

Per  the  p-value  of  less  than  0.0001,  Hypothesis  5  is  supported.  The  JMP-IN  4 
output  page  for  an  analysis  of  variance  of  these  constructs  is  displayed  in  Figure  9. 
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Figure  9.  JMP  Output:  One-Way  Analysis  of  Definite  Answer  Rate  By 
Relevance  (Definite  Answer  Rates  for  Each  Degree  of  Relevance) 


Oneway  Anova 
Summary  of  Fit 
Rsquare  0.042285 

Adj  Rsquare  0.037897 

Root  Mean  Square  Error  0.478073 
Mean  of  Response  0.387244 
Observations  (or  Sum  Wgts)  878 


Source  DF 
Relevance  4 
Error  873 
C.  Total  877 


Analysis  of  Variance 

Sum  of  Squares  Mean  Square  F  Ratio  Prob  >  F 
8.80963  2.20241  9.6363  <.0001 

199.52750  0.22855 

208.33713 


Means  for  Oneway  Anova 


Level  Number 

Mean 

Std  Error  Lower  95% 

Upper  95% 

1 

107 

0.308411 

0.04622 

0.21770 

0.39912 

2 

133 

0.330827 

0.04145 

0.24947 

0.41219 

3 

214 

0.266355 

0.03268 

0.20221 

0.33050 

4 

251 

0.458167 

0.03018 

0.39894 

0.51739 

5 

173 

0.526012 

0.03635 

0.45467 

0.59735 

Std  Error 

uses  a  pooled  estimate  of  error  variance 
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H6:  Relevance  associates  with  Familiarity 

This  hypothesis  suggests  that  relevance  will  associate  with  familiarity,  so  that  as 
levels  of  relevance  change,  levels  of  familiarity  and  will  change  significantly  as  well.  The 
ANOVA  provided  the  following  results:  F(4,867)  =  111.56;  p  <  0.0001,  which  indicates 
strong  support  for  the  association  between  relevance  and  familiar,  and  therefore  strongly 
supports  hypothesis  six. 

Closer  inspection  of  the  data  via  Tukey-Kramer  analysis  revealed  that  there  were 
significant  differences  in  mean  familiarity  rating  among  the  all  but  one  pairing  of 
relevance  rating.  This  further  supports  Hypothesis  6.  Visually,  the  relationship  seems  to 
display  a  direct  and  proportional  relationship. 

Hypothesis  6  is  supported.  The  JMP-IN  4  output  page  for  an  analysis  of  variance 
of  these  constructs  is  displayed  in  Figure  10. 
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Figure  10.  JMP  Output:  One-Way  Analysis  of  Familiarity  By  Relevance  (Mean 
Familiarity  Rate  for  Each  Degree  of  Relevance) 


Oneway  Anova 
Summary  of  Fit 
Rsquare  0.339791 

Adj  Rsquare  0.336745 

Root  Mean  Square  Error  1 .060348 
Mean  of  Response  2.604358 
Observations  (or  Sum  Wgts)  872 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio  Prob  >  F 
Relevance  4  501.7023  125.426  111.5550  <.0001 

Error  867  974.8012  1.124 

C.  Total  871  1476.5034 


Means  for  Oneway  Anova 

Level  Number  Mean  Std  Error  Lower  95%  Upper  95% 


1 

108 

1.47222 

0.10203 

1.2720 

1.6725 

2 

131 

1.81679 

0.09264 

1.6350 

1.9986 

3 

214 

2.30374 

0.07248 

2.1615 

2.4460 

4 

248 

2.97581 

0.06733 

2.8437 

3.1080 

5 

171 

3.76023 

0.08109 

3.6011 

3.9194 

Std  Error  uses  a  pooled  estimate  of  error  variance 
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Hypothesis  Summary 

A  summary  of  analysis  results  for  each  hypothesis  is  shown  in  Figure  1 1 . 
Hypotheses  2,  4,  5,  and  6  are  supported.  Hypotheses  1  and  3  were  not  supported. 


Figure  11.  Table  of  Results:  Hypothesis  p-values,  n,  and  support 


Hypothesis 

P 

n 

Supported 

HI 

Familiarity  Positively  Associates  with  Accuracy 

0.3302 

870 

No 

H2 

Definite  Answers  Positively  Associate  with  Accuracy 

0.0009 

884 

Yes 

H3 

Relevance  Positively  Associates  with  Accuracy 

0.1957 

878 

No 

H4 

Familiarity  Positively  Associates  with  Definite  Answers 

<0.0001 

870 

Yes 

H5 

Relevance  Positively  Associates  with  Definite  Answers 

<0.0001 

878 

Yes 

H6 

Relevance  Positively  Associates  with  Familiarity 

<0.0001 

872 

Yes 

It  can  be  seen  in  Figure  1 1  that  the  n  was  not  the  same  for  each  hypothesis  test. 
This  is  a  result  of  missing  answers  on  the  questionnaires  and  JMP-IN  4’s  automatic 
exclusion  of  data  points  that  do  not  consist  of  a  matched  pair  of  data  for  a  given  test. 
Since  37  participants  were  shown  24  questions,  an  n  of  888  would  be  ideal.  However,  in 
the  case  of  Hypothesis  1  for  example,  an  n  of  only  870  was  used.  This  means  that  18 
data  points  were  excluded  because  participants  omitted  an  answer  for  familiarity,  a 
manipulation  answer,  or  both. 

It  is  interesting  to  note  that  in  the  model,  potential  relations  between  three 
constructs  and  accuracy  are  illustrated.  Of  three  items  that  were  thought  to  associate  with 
accurate  answers,  only  the  definite  answers  seem  to  actually  do  so  to  a  statistically 
significant  level. 

Familiarity  did  associate  with  definite  answers,  and  relevance  associated  with 
both  familiarity  and  definite  answers.  Intuitively,  one  might  expect  in  such  a  case  that 
the  familiarity  and  relevance  would  have  also  directly  associated  with  accurate  answers. 
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Obviously,  the  data  did  not  support  that.  The  means  and  variances  charts  that  JMP 
produces  as  part  of  the  ANOVA  was  examined  for  visual  trends  in  terms  of  how 
familiarity  and  relevance  may  have  associated  with  accuracy. 

While  nothing  that  is  statistically  significant  can  be  gleaned  from  observing  the 
means  and  variances  charts  directly,  there  appeared  to  be  a  relatively  flat  response  among 
the  different  degrees  of  familiarity  and  relevance  and  their  corresponding  rates  of 
accuracy.  There  was  a  slight,  but  visually  perceptible  positive  trend  between  familiarity 
and  accuracy,  but  with  a  slightly  negative  trend  between  relevance  and  accuracy.  No 
intuitive  reason  can  be  stated  for  this  at  this  time,  and  this  enigma  may  serve  as  a  topic 
for  future  research. 

Additional  Feedback 

Feedback  given  by  a  number  of  participants  indicated  that  the  WIGO  messages 
were  not  read  by  all  persons  who  receive  them.  Further,  there  were  indications  that 
participants  obtain  information  on  a  number  of  the  topics  by  means  other  than  the  WIGO 
messages,  including  personal  experience  and  the  news  media. 
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Chapter  5  -Discussion 


Findings 

The  results  of  the  analysis  are  both  surprising  and  interesting  because  of  which 
hypotheses  were  supported  by  the  data,  and  which  were  not  supported.  Intuitively,  a 
positive  association  was  expected  between  each  construct  and  each  other  construct, 
however  this  did  not  turn  out  to  be  the  case.  The  lack  of  support  for  the  associations 
between  accuracy  rate  and  participant-rated  familiarity  and  relevance  was  probably  the 
most  surprising  finding  in  this  study. 

It  was  thought  that  higher  perceived  familiarity  with  a  topic  might  make  the 
participants  more  likely  to  recognize  a  manipulation  to  the  information.  Intuitively,  there 
may  be  a  number  of  contributing  factors  that  underlie  a  person’s  determination  of 
whether  they  are  familiar  with  something.  The  single  participant-provided  rating  for 
familiarity  was  used  for  simplicity  in  this  exploratory  study.  Verbal  feedback  given  by 
participants  about  the  experiment  indicated  that  a  number  of  persons  did  not  regularly 
read  the  WIGO  E-mail  messages,  even  though  there  are  delivered  to  all  AFIT  students. 
In  these  cases,  their  familiarity  with  the  topic  would  obviously  due  to  means  other  than 
reading  the  WIGO  message. 

It  was  also  surprising  that  relevance  of  the  information  to  the  person,  as  rated  by 
the  participant,  did  not  correlate  with  rate  of  accuracy  either.  It  was  conceptualized  that 
relevance  might  have  an  effect  on  accuracy,  possibly  because  of  an  increased  personal 
interest  in  the  information.  This  concept  might  be  expanded  or  controlled  in  future 
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research  to  channel  a  person  to  consider  there  answer  from  a  certain  perspective,  such  as 
“relevant  to  me  in  the  perfonnance  of  my  job”  or  “relevant  to  me  on  a  personal  level.” 

The  hypothesis  regarding  an  association  between  relevance  and  familiarity  was 
supported.  By  examining  the  results  of  the  analysis  of  variance  that  was  obtained  via 
JMP,  there  was  strong  support  for  a  direct  association  between  these  constructs.  Further, 
it  also  appears  that  familiarity  is  generally  at  a  slightly  lower  level  than  the  relevance 
rating  that  a  participant  gives.  While  no  causality  can  be  detennined  by  this,  determining 
the  nature  of  the  relationship  between  relevance  and  familiarity  (i.e.  does  relevance  lead 
to  familiarity?)  is  an  area  that  can  be  examined  in  future  research.  The  findings  of 
support  for  associations  between  familiarity,  relevance,  and  definite  answers  matched 
intuitive  expectations. 

Overall,  these  findings  are  significant  because  they  go  against  intuition  that  a 
familiar  person,  or  a  person  to  whom  the  information  is  more  relevant,  is  more  likely  to 
accurately  identify  whether  manipulation  was  present.  The  amount  of  relevance  and 
familiarity  (or  both,  as  in  the  case  where  someone  might  be  considered  a  subject  matter 
expert)  did  not  directly  associate  with  accuracy.  These  constructs  did  however  associate 
with  the  rate  of  definite  answers  given,  and  in  turn,  definite  answers  did  associate  with 
increased  accuracy.  What  this  means  is  that  a  person  who  is  a  subject  matter  expert  and 
who  is  willing  to  give  a  definite  answer  may  be  the  most  accurate  determiner  of  whether 
manipulation  is  present.  Lacking  the  commitment  to  a  definite  answer,  a  person  who  is  a 
subject  matter  expert  is  no  more  accurate  than  a  person  who  is  unfamiliar  and/or  who 
perceives  little  relevance. 
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The  results  of  this  experiment  leave  question  whether  a  person  with  lower 
familiarity  and  relevance  ratings  for  given  information  might  examine  the  information 
differently  than  a  person  that  is  familiar  and  does  find  it  to  be  relevant.  Conversely, 
perhaps  a  person  who  feels  the  information  is  familiar  and/or  relevant  might  exercise  less 
care  in  making  their  choice.  A  grim  saying  in  military  circles  regarding  lethal 
misfortunes  associated  with  complacency  is  “Complacency  Kills.”  This  raises  a  question 
about  vigilance  and  suspicion  and  how  they  might  relate  to  conditions  of  familiarity  and 
relevance. 

Limitations 

The  volunteers  in  the  study  are  comprised  of  a  convenience  sample.  This 
provided  an  accessible  pool  of  volunteers  that  function  in  a  military  environment,  which 
is  administrative  in  nature;  this  does  in  part  match  part  of  the  Air  Force  Office  of 
Scientific  Research’s  intent  to  gain  knowledge  about  deceit  in  military  operations. 
However,  there  are  a  variety  of  environments  in  which  military  operations  occur,  and  the 
results  of  this  particular  study  may  not  necessarily  translate  to  a  combat  environment,  or  a 
different  type  of  information  (sensitive  real-time  intelligence  information,  for  example), 
or  an  organization  with  different  characteristics  (such  as  different  size,  culture, 
information  dissemination  characteristics,  etc.).  Future  research  would  need  to  evaluate 
differences  between  the  environment  studied  here  and  the  scenario  that  is  examined  at 
that  time. 

As  an  exploratory  study,  the  constructs  that  were  measured  were  general  in 
nature.  Although  the  ties  between  the  constructs  of  familiarity  and  relevance  of  the 
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information  used  in  the  experiment  are  important  to  the  results  for  this  study,  a  future 
study  may  explore  these  items  from  multiple  different  perspectives  to  gain  more 
knowledge.  Perhaps  the  overall  topic  and  the  facts  within  the  passage  could  be  evaluated 
separately. 

The  infonnation  pool  that  was  used  was  also  selected  for  convenience  in  the 
chosen  participant  pool  environment.  The  experimental  design  aimed  to  use  information 
that  had  previously  been  exposed  to  the  target  audience;  however,  this  distribution  did  not 
necessarily  mean  that  it  was  read  by  the  participants.  As  was  found  in  feedback  given 
during  the  experiment,  persons  may  not  read  the  information  from  the  WIGO,  but  may  in 
fact  obtain  their  infonnation  on  the  same  topics  from  alternate  sources.  The  infonnation 
that  was  obtained  from  these  alternate  sources  may  have  been  presented  differently  than 
(or  disagreed  with)  the  WIGO  messages.  This  may  obviously  influence  the  results  of  this 
experiment.  In  addition,  from  a  longitudinal  perspective,  the  information  pool  may  in 
some  cases  have  been  overcome  by  real  world  events,  including  subsequent  changes  in 
the  items  that  are  represented  in  the  message,  distortion  of  memories  about  the  real 
events,  and  perhaps  similarity  to  more  recent  events  causing  confusion  of  two  different 
items. 

More  specifically,  in  regard  to  the  text  questions  that  were  chosen  for  use  in  the 
experiment,  they  were  examined  by  the  author  of  the  experiment  and  selected  with  a 
rationale  that  they  would  be  applicable  to  AFIT  students.  Further,  the  types  of 
manipulation  selected  for  use  in  the  experiment  dictated  which  messages  were  potentially 
useful,  and  further  narrowed  the  pool  of  eligible  messages.  (As  explained  in  Chapter  3,  a 
message  had  to  be  manipulable  by  one  of  the  four  types  of  manipulation  that  were 
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focused  on  for  the  experiment;  this  excluded  any  messages  that  were  manipulable  only  by 
other  methods). 

Reading  textual  infonnation  in  any  significant  volume  can  be  time  consuming. 
This  reduced  the  number  of  questions  that  could  feasibly  be  asked  during  an  experiment 
session  in  order  to  keep  the  length  of  time  required  of  each  participant  to  a  reasonable 
amount  (less  than  an  hour).  As  such,  the  low  number  of  questions  may  have  allowed 
question-specific  anomalies  to  have  greater  potential  to  influence  results  than  an 
experiment  with  a  larger  cross-section  of  questions. 

The  low  number  of  questions  that  were  included  in  the  experiment  also  limited 
the  number  of  manipulation  types  that  were  used,  and  this  limited  the  depth  to  which  each 
manipulation  type  could  be  tested  and  analyzed  (i.e.  varying  degrees  of  manipulation). 
As  an  exploratory  study,  the  variety  provided  opportunities  for  more  things  to  be  initially 
checked  in  hopes  of  guiding  future  research,  however  this  reduces  the  ability  for  the  each 
type  to  be  thoroughly  checked,  and  inhibits  any  subtle  trends  from  emerging  to  a  point  of 
statistical  significance. 

Implications  for  Research 

Above,  support  was  found  for  association  between  familiarity,  relevance,  and 
definite  answers.  Support  was  also  found  for  an  association  between  definite  answers 
and  higher  accuracy  rates.  However,  no  support  was  found  for  a  direct  association 
between  level  of  familiarity  and/or  relevance  to  accuracy  rates. 

A  further  study  on  this  topic  might  utilize  a  setting  where  more  participants 
provide  information,  and  more  things  can  be  manipulated  and  controlled;  an  example 
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might  be  an  Air  Force  battle  lab  or  training  environment,  where  real-time  information  is 
used,  and  the  results  of  implementation  of  those  decisions  could  be  examined  after 
numerous  trials.  Such  an  environment  would  allow  for  detailed  development  of  an 
experiment  and  instrument  to  an  extent  that  was  not  possible  in  the  AFIT  used  within  this 
study. 

The  experiment  allowed  participants  to  give  a  “definite  answer”  or  an  answer  that 
was  not  worded  with  “definitely”  in  it.  This  was  a  binary  decision  in  terms  of  “definite 
answer”  and  participants’  personal  thresholds  for  committing  to  a  definite  answer  may 
have  caused  some  variance  that  is  not  accounted  for  in  this  experiment.  This  concept 
could  be  more  closely  examined  in  future  research  to  determine  if  better  choices  for 
confident  answers  can  be  presented.  In  addition,  incentives  for  giving  a  definite  answer 
were  not  given.  This  and  its  implications  may  be  studied  more  closely  in  the  future. 

Suggestions  for  Future  Research 

Future  research  is  needed  in  this  area.  It  should  expand  on  the  research 
accomplished  here  and  in  extant  literature  in  number  of  ways.  The  first  might  be  a  more 
detailed  study  of  the  hypotheses.  Perhaps  familiarity  and/or  relevance  could  be  studied  to 
a  finer  degree  of  detail.  In  addition,  the  intuitive  question  arose  that  asked  whether 
relevance  of  the  information  to  a  person  somehow  influenced  or  caused  a  person  to 
become  familiar  with  that  information. 

The  distortion  of  infonnation  was  the  focal  point  for  this  study.  The  work  here 
can  be  extended  to  cover  other  information  manipulation  strategies  such  as  concealment 
described  by  Turner,  et  al,  (1975)  and  the  other  deceptive  tactics  that  are  presented  by 
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Johnson,  et  al  (1993),  and  Biros,  et  al  (2002).  This  area  is  ripe  for  the  study  of  other 
strategies  as  well. 

Implications  for  Managers 

In  cases  where  infonnation  is  obtained  mostly  by  reading  it  (as  opposed  to 
persons  speaking  it,  or  viewing  recorded  media),  it  is  possible  that  the  lack  of  cues  could 
lead  persons  to  believe  that  a  sense  of  familiarity  with  or  relevance  to  the  information 
helps  them  detennine  whether  the  information  has  been  manipulated  or  not.  This  sense 
of  familiarity  and/or  relevance  was  not  supported  to  be  helpful  in  making  an  accurate 
determination  of  manipulation  in  this  study. 

The  key  item  was  whether  the  person  was  willing  to  commit  to  a  definite  answer. 
It  is  possible  that  a  person  would  take  into  account  factors  that  were  not  studied  in  this 
experiment  when  assessing  the  presence  of  manipulation.  If  these  other  factors  conflicted 
with  an  intuitive  conclusion  that  might  be  drawn  from  degree  of  familiarity  or  relevance, 
the  analysis  here  could  support  the  lack  of  a  definite  answer  negates  any  perceived 
confidence  that  relevance  or  familiarity  alone  might  offer.  This  may  allow  potentially 
more  detenninistic  factors  to  be  more  heavily  considered  in  the  assessment  of  the 
information. 

Increases  in  the  use  of  automated  systems  where  text  information  is  used  may 
increase  the  need  for  knowledge  in  the  area  of  detecting  when  the  information  contained 
within  it  has  in  fact  been  manipulated. 
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Closing 

I  am  grateful  for  the  opportunity  to  have  participated  in  graduate  study  at  the  Air 
Force  Institute  of  Technology  and  in  particular  for  the  opportunity  to  research  this 
fascinating  topic.  Some  knowledge  has  been  discovered,  however,  it  is  only  a  very  small 
part  of  the  knowledge  that  is  needed  to  improve  practical  performance  of  detecting  the 
deception  that  threatens  our  information-dependent  military  forces.  It  is  my  hope  that 
this  exploratory  study  stimulates  future  research  in  this  area  and  that  the  proceeds  of  that 
work  will  truly  enhance  our  capability  to  protect  our  national  interests. 
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